Analogy completion has been a popular task in recent years for evaluating thesemantic properties of word embeddings, but the standard methodology makes anumber of assumptions about analogies that do not always hold, either in recentbenchmark datasets or when expanding into other domains. Through an analysis ofanalogies in the biomedical domain, we identify three assumptions: that of aSingle Answer for any given analogy, that the pairs involved describe the SameRelationship, and that each pair is Informative with respect to the other. Wepropose modifying the standard methodology to relax these assumptions byallowing for multiple correct answers, reporting MAP and MRR in addition toaccuracy, and using multiple example pairs. We further present BMASS, a noveldataset for evaluating linguistic regularities in biomedical embeddings, anddemonstrate that the relationships described in the dataset pose significantsemantic challenges to current word embedding methods.
展开▼
